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Editor’s Note 

The papers in this Special Issue were all peer-reviewed, debated, formally discussed and revised 
before, during and after the International Consultation on Meta-Analysis held at Potsdam, 
Germany in March of 1994. The articles and formal discussions were all taken into account in the 
preparation of the final contribution of the series. Methodological Guidelines for Systematic Reviews 
of Randomized Controlled Trials in Health Care. The Guidelines are just that, only guidelines. They 
are not advanced as a maximum standard, nor as an inflexible minimum threshold of methodologi¬ 
cal standards. They do suggest, however, what the current understanding of the state of science in 
overview methodology requires for mcta-analytically derived evidence to be deemed admissible or 
credible. Moreover, given that the methodology is young, the Guidelines should be considered as a 
milestone rather than a goal achieved. We had hoped to have two sets of guidelines in this issue, one 
for experimental trials and a second one pertaining to observational epidemiological studies. The 
challenges in reaching a consensus about the second set were too formidable to overcome during the 
Potsdam Consultation itself and even during the few months following the meeting, before we bad 
to go to press. They will be published in a subsequent issue. 

The participants at the Potsdam International Consultation on Meta-Analysis were only those 
who are authors of presentations and discussions included in these proceedings. We thank diem all 
for the rigour of the material and the timeliness with which first drafts and subsequent revisions 
were submitted. That has allowed early publication and thus a more useful resource to the scientific 
community. 

Sponsors of the Potsdam Consultation are listed overleaf. 

Support for the publication of this Special Issue was given by the Federal Institute for Drugs and 
Medical Products of Germany with an Accountable Contribution for the support of scientific 
development. 


Walter O. Spitzer 
Editor 
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“THE CHALLENGE OF META-ANALYSIS”: DISCUSSION. 
INDICATIONS AND CONTRA-INDICATIONS FOR 
META-ANALYSIS 


NORBERT VICTOR 

Institut ffir Med. Biometric und Infonnttik, Ruprecht-Karli Univmillt Heidelberg, 
Im Neueoheuner Feld 305, 69110 Heidelberg, Germany 


INTRODUCTION 

Walter Spitzer gave us a general and compre¬ 
hensive review of the challenges of meta- 
analysis in the keynote address [1]. Therefore, 
my comments will also be general. His task was 
to present the aims of the conference and to 
point out the main difficulties of meta-analysis. 
His speech reflected scepticism about meta¬ 
analysis, but also a growing realization that 
meta-analysis is needed and is valid in empirical 
research. 

IS META-ANALYSIS THE SIGHT TERM? 

ASE META-ANALYSES OBSERVATIONAL? 

First, let me discuss the two questions which 
Spitzer considered of “minor importance”. 

(i) Should we continue to use the term meta¬ 
analysis? I think we should 

Aggregation of information is done in differ¬ 
ent ways, ranging from collecting data in a 
single pre-planned experiment to unstructured 
reviews of existing literature. In this spectrum 
of aggregation procedures, there are gradual 
transitions: 

• between multi-centre studies and meta- 
analyses of similar parallel conducted studies, 
or 

• between meta-analyses for a complex of prob¬ 
lems and syntheses of evidence of the type 
mentioned in the keynote address. 

I believe in this continuum the procedure 
“to analyse the results of Studies concurrently, 
according to a pre-specified protocol, where the 
results are also derived by pre-specified analyses " 
deserves its own name. Meta-analysis seems to 


be the right term for this procedure. However, 
I am against the use of this term for other kinds 
of aggregation of information. Hence, the given 
definition is in some aspects more stringent than 
the definition of Huques [2] quoted by Spitzer in 
his 1991 editorial [3J. However, in general terms, 
Huques’ definition is acceptable to me because 
it allows only the integration of “combinafale” 
studies in a meta-analysis. 

Thus, during this conference, we should dis¬ 
miss the classification and demarcation of 
the different kinds of procedures of combining 
information: overviews, systematic reviews, syn¬ 
theses of evidence, meta-analyses etc. and clarify 
their differences, when they exist. 

(UJ Does meta-analysis of experimental trials 
become observational? My answer here is also a 
clear “YES" 

The experimental approach is based on the 
pre-spedfication of the conditions of conduct of 
a trial including a priori sample size calculations. 
Such requirements are never fulfilled in meta- 
analyses. We find (observe) various experimen¬ 
tal conditions of the trials to be aggregated and 
have to accept them as they are or have to 
exclude the trials. The number of patients in¬ 
cluded in a meta-analysis is determined by the 
set of existing trials, the searching procedure 
and the inclusion criteria of the protocol for 
the meta-analysis. Therefore, meta-analyses are 
always observational! 

Consequently, the,confidence we can place in 
results of meta-analyses cannot be the same as 
that we place in the results of controlled trials, 
which have attained their objectives. It can only 
be comparable to the confidence in the results of 
observational trials. 
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INDICATIONS AND CONTBA-INDICATIONS 
OF META-AN AJLYSES 

Let me now comment on Spitzer’s earlier 
doubts about the necessity of meta-analyses and 
on the path he followed to become convinced 
that they are unavoidable. 

Aggregation of information is indispensable for 
empirical research 

As scientists, we are often obliged to support 
urgent official administrative decisions and in 
such situations we are obliged to use all existing 
information in the best possible way. The best 
way varies, depending on the problem at hand, 
the level of knowledge about the problem and 
the format of the existing information. In some 
cases, the best way is meta-analysis. In other 
situations, other procedures of aggregation of 
information can be adequate: systematic re¬ 
views, syntheses of evidence, or—if there is no 
time pressure—a new experimental trial Paren¬ 
thetically, I would like to give my opinion that 
syntheses of evidence are a kind of meta¬ 
analysis with special inclusion criteria and 
distinctive methods of imputing weights on the 
data. 

Let me try to justify the necessity of aggregat¬ 
ing strategies. Usually, before tackling a prob¬ 
lem, scientists collect all existing information by 
searching the literature. Only a lack of knowl¬ 
edge concerning a problem justifies further in¬ 
vestigation of it. So, every scientific paper begins 
with an overview of the literature. Hence, a 
meta-analysis alone can also be regarded as an 
introduction to a venture without continuation. 
Consequently, meta-analyses in general are 
necessary because they are sometimes the best 
way to combine existing information; but not 
all meta-analyses that we find in our journals 
today are necessary. There is widespread misuse 
of meta-analysis and often they are carried 
out inadequately in situations where they are 
inappropriate. 

Consequently, we should first define rules 
when meta-analyses arc an appropriate instru¬ 
ment and when not. Therefore, I propose to 
launch Walter Spitzer’s “Potsdam Standards” 
with a paragraph proposing indications and 
contra-indications for meta-analysis. 

Proposed indications for meta-analysis 
are: 

• The necessity of an urgent decision, when lack 

of time renders the conduct of a new trial 

impossible 


• Research on safety aspects of drugs and 
other therapies especially the evaluation of 
side-effects (when one searches for adverse 
side-effects the exclusion of any study is 
inappropriate) 

• The existence of many non-condusive studies 
on treatment, where small effects are import¬ 
ant and when target reaching trials are unreal¬ 
istic in view of needed sample size and time 

• Contradicting results of studies or effects 
which vary too much among different types of 
subjects can also be an indication for meta¬ 
analysis. Then global statements for several 
studies or even one study can be senseless and 
the result of an (explorative) meta-analyses 
taking co-factors into account can provide a 
more detailed statement. 

Contra-indications 

• Contra-indications include meta-analyses as 
the basis of drug approval and registration. It 
is a task for the regulatory agency to do a 
synthesis of evidence for the studies included 
in the file 

• Contra-indicated is the attempt to make an 
irrelevant or unimportant effect “significant” 
by combining a lot of insignificant studies 
showing small effects. This is misuse of 
meta-analysis! 

• It is also a misuse of meta-analysis if a 
researcher uses it to avoid the hard work of 
conducting a study of his own. A researcher 
must always be stimulated by a problem and 
not by the possibility of increasing the number 
of his publications. However many of the 
meta-analyses which fill journals today seem 
to be stimulated by the expectation of rapid 
publication by a meta-analysis. It seems that 
several reports about similar studies lead 
unavoidably to a meta-analysis today. Such 
activities do really deserve Spitzer’s term 
“second class research”. One example: I have 
just seen a manuscript submitted to a journal, 
presenting a meta-analysis of nine trials of the 
effects of evening primrose oil in the treatment 
of pre-menstrua! syndrome. All trials show 
irrelevant effects and—with one exception— 
they are all of questionable quality. 

The main danger of meta-analyses is the 
failure to recognize these contra-indications. 
Studies which are too small are carried out in 
the hope of, and with the hint of, inclusion in 
future meta-analyses. To avoid this danger, it 
should be a pre-requisite that before starting a 
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meta-analysis, its need must be examined; the 
need must then be justified in the publication. 

HETEBOGENEmES IN META-ANALYSES 

I missed the mention of one important 
problem of meta-analysis in Spitzer's opening 
address [I], and also in his 2991 editorial (3): the 
interpretation of heterogeneous results of meta- 
analyses. There has been a lot of discussion in 
the literature about this problem, e.g. Ref. [4], 
Nevertheless, beyond all proposals on how to 
handle heterogeneity we must propagate, as a 
highest precept results of a meta-analysis cm 
only be Interpreted if existing heterogeneities 
am be adequately explained by methodological 
heterogeneities. 

We must avoid forcing together a hetero¬ 
geneous pattern of results into a meaningless 
overall conclusion; especially, but not only, 
senseless global estimates must be avoided. On 
the other hand, we will always need statements 
about the size of the effects to judge their 
relevance, because this cannot be done by a 
p-value, or merely by the determination of 
“significance”, cf. Ref. [5]. Therefore, in the 
protocol we need pre-specification of the mini¬ 
mal relevant effect and the maximal degree of 
heterogeneity that will permit global statements. 
Often only detailed statements make sense as 
the result of a meta-analysis, or the declaration: 
“A genera] statement is not possible". In the 
“Potsdam Standards" we need rules to clarify 
under what circumstances different kinds of 
global statements are allowed. 

Let me give an example where the cause of 
heterogeneity in some published meta-analyses 
was explained: if a set of studies on low molecu¬ 
lar weight hepariss vs standard heparins is split 
into double blind and single blind/open studies, 
we get a positive effect for wound haematoma in 
one group of studies, and a negative one in the 
other. 

Remark from a personal view: Normally, we 
exclude studies of questionable quality from our 
meta-anaiyses. Nevertheless, I always compare 
the results of the meta-analysis (of the “good 
studies”) with the results of the “bad” studies. 
If this comparison shows remarkable differ¬ 
ences, I am very careful in interpreting the result 
of the meta-analysis of acceptable studies. 

SOME FUKTHEJt COMMENTS 

Personally, I very strongly support Spitzer's 
proposal that we keep meta-analyses of exper¬ 


imental trials and of observational studies in 
different compartments. Surely we need more 
detailed guidelines for the second kind of meta- 
analyses, because the pitfalls and dangers in 
this case are much more numerous than in 
the former. Furthermore, experience with such 
meta-analyses is very limited; personally, I do 
not have any experience of such meta-analyses. 

I agree with our keynote speaker that, in our 
current understanding of methodology, accept¬ 
able meta-analyses of observational studies can 
only be done in a few exceptional cases. On the 
other hand, important decisions often demand 
information which is only available from obser¬ 
vational studies. Therefore, we must uy to 
smooth the path for meta-analyses in this field. 
I agree with Spitzer that this undertaking is very 
difficult, but it is not hopeless. 

Therefore, I propose that our discussion over 
the next 2 days should concentrate on the 
problems of meta-analysis of observational 
studies. There is a serious lack of guidelines in 
this field, in contrast to the practice of aggregat¬ 
ing controlled trials. We have rules for the 
assessment of controlled trials (e.g. (6-8])- 
We have both implicit (e.g. [9]) and explicit 
(e.g. (10)) guidelines for meta-analysis. I refer 
especially to the extensive guidelines of the 
Versailles Conference, chaired by J. F. Boissel, 
and mentioned in Ref. [10]. In my opinion, 
those are the best ones—perhaps a little too 
detailed—and I regret that they are not well- 
known. They are well-suited as the paint of 
departure for the “Potsdam Standards”. 

A FINAL COMMENT 

The elaboration of guidelines is one way to 
improve quality and reliability of meta-analyses. 
Another task for the scientific community 
should be an effort to establish registers of 
studies. If ail studies were registered before their 
implementation, the greatest danger of a meta¬ 
analysis, publication bias, could be avoided. 
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